What Stuck With Me: Notes from Learning Claude

Lately I've been learning more about how Claude and generative AI actually work, and like I do with everything I learn, I kept notes. Lots of them.

These are the ideas that stuck with me most, the ones that changed how I use AI day to day, both as a Salesforce developer building with Agentforce and as someone who now uses Claude for almost everything.

If you're just starting with AI, or you use it daily but have never stopped to ask why it behaves the way it does, I hope a few of these land for you too.

1. It's writing one word at a time

The first big idea: generative AI has four core properties.

1. Next token prediction
2. Knowledge
3. Working memory
4. Steerability

The one everything else builds on is next token prediction. The model writes its answer word by word, based on what tends to follow what in the data it learned from.

That sounds simple, but it explains a lot. When a task looks like something the model has seen many times (summarising, reformatting, explaining a common concept), it's brilliant. When the task is new or rare, or needs it to tell "true" apart from "sounds true", that's where it struggles.

My takeaway

The output was generated, not looked up. Once you hold on to that, you know exactly what kind of checking to do.

2. How an AI assistant is made

Modern AI assistants go through two stages of training:

1. Pretraining: the model learns patterns from massive amounts of data.
2. Fine-tuning: that raw model is refined towards broadly safe and helpful behaviour. This is the step that turns it into an assistant.

I found this useful because both stages leave fingerprints. Pretraining shapes what the model knows. Fine-tuning shapes how it behaves, including some of its habits that aren't always helpful.

3. The "shadow areas" worth knowing about

This was one of my favourite parts. Fine-tuning brings some side effects, and naming them makes them much easier to spot:

• Sycophancy: people tend to prefer agreeable answers, so the model can lean towards telling you what you want to hear.
• Verbosity: it defaults to longer answers than you often need.
• Overcaution: when safety training leans conservative, the model may hedge heavily or refuse requests that are actually fine.
• Loose confidence calibration: confidence is hard to train, so how sure it sounds isn't always how right it is.

Now when I get a long, overly agreeable answer, I don't just accept it. I push back, ask for the short version, or ask it to argue the other side.

4. Where made-up answers like to hide

This line from my notes is the one I repeat most: fabrication concentrates in specificity.

Names, dates, statistics, citations, URLs, quotes. The more precise a claim is, the more it deserves a second look. A general explanation of how Flows work is low risk. A specific release note, limit or API version number is exactly where I verify.

The good news is that a lot of product features exist to push this limit further out:

• Citations: so you can see what's backed by a source and what's generated.
• Trained uncertainty: the model saying "I'm not sure about that" instead of guessing.
• Constrained generation and skills: narrowing the space where fabrication can sneak in.
• Generator-verifier loops: one step produces the output, another checks it.

If you've built with Agentforce, that last one will feel familiar. Grounding and guardrails are the same idea: don't let the model free-write where it shouldn't.

5. What it knows, and what it doesn't

AI is strong on mainstream science, popular programming languages and well-known history. It's weaker on:

• rare or niche topics
• events after its training cutoff
• specialised domains
• local knowledge

Tools help close these gaps: web search, MCP connectors, tool use, and models being upfront about their knowledge cutoff.

The practical tips I wrote down:

1. Verify anything time-sensitive.
2. Test before you trust it in a new domain.
3. Watch for default assumptions.
4. Use the tools that are available.

For us in the Salesforce world this one is real. Three releases a year means "current" changes fast, so I always double-check feature names and availability.

6. More context isn't always better

This one surprised me the most.

My instinct used to be: give the AI everything. Paste the whole document, every message, all the context I can find. More information means better answers, right?

Not always. Think of cramming for an exam. There's a limit to how much you can hold in your head at once, and the stuff in the middle tends to slip away first.

Models behave similarly. Instructions at the start and end of a long context get followed; the middle gets buried. A well-known research paper, Lost in the Middle (Liu et al., 2023), found that models were noticeably less accurate when the key information sat in the middle of a long context rather than near the beginning or end.

My takeaway

Share what's relevant, not everything, and put the most important instruction where it won't get lost.

7. Steerability: the gap between what you said and what you meant

Steerability is the model's ability to follow your direction. And here's the bit that clicked for me: it follows your instructions the same way it does everything else, by continuing a pattern.

That makes it very steerable. It also means there's always a gap between what you meant and what landed, and most of the interesting failures live in that gap.

• Works well: short, concrete, checkable instructions. Format specs, length limits, clear roles.
• Struggles with: long chains of reasoning, vague or abstract instructions, and anything needing exact numbers or logic.

Two failure types to watch for:

• Reasoning drift: small errors add up over a long task.
• Letter over spirit: it followed the instruction, but missed the intent.

System prompts, code execution, visible reasoning and structured outputs all exist to keep your intent from getting diluted.

My favourite tip

When an instruction is followed literally but uselessly, restate the goal. Repeating the same instruction louder won't close the gap.

8. The 4D framework that ties it together

The courses bring everything together in the 4D framework:

• Delegation: deciding how to split work between you and AI.
• Description: communicating clearly what you want from the AI.
• Discernment: thoughtfully and critically evaluating what it gives you.
• Diligence: taking responsibility for how you use AI.

In day-to-day work, you'll mostly live in the inner loop of Description and Discernment: ask clearly, check whether the result meets your expectations, adjust, repeat.

What I liked is how the earlier ideas map onto it:

• Next token prediction is the foundation of Discernment. Knowing the output was generated tells you what to scrutinise.
• Steerability is what makes Description powerful, and also what limits it. Understanding the gap between words and intent changes how you write prompts and where you add checkpoints.

What I'm taking with me

If I had to boil it all down:

• Treat AI output as generated, not looked up. Check the specifics.
• Give focused context, not all the context.
• When it misses the point, restate the goal instead of repeating yourself.
• Stay in the loop: describe, check, refine.

I'm still working through more courses, so I'll keep adding to this as I go. If you're learning too, I'd love to hear what's stuck with you. Drop it in the comments or find me on LinkedIn.